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Ratings developed by 0PM psychologists were established for the ACWA 
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through the Behavioral Consistency Method* Validity issues in the 
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ADAPTING ASSESSMENT PROCEDURES FOR DELIVERY 
VIA AN AUTOMATED FORMAT^ 



INTRODUCTION 

Based on recommendations from the National Performance Review, the General Accounting 
Office, and the Merit Systems Protection Board, as well as feedback from agencies on the 
current hiring process, the Office of Personnel Management (0PM) decided to explore 
alternative examining procedures for positions covered by the Administrative Careers with 
America (ACWA) examination. (See Table 1 for the list of occupations covered under the 
ACWA program.) One requirement for these new procedures was that they be automated for 
use with OPM’s recently developed Microcomputer Assisted Rating System (MARS). MARS 
is a highly efficient delivery system for presenting staffing options to agencies such as rating 
schedules, tests, and sample questions for structured interviews. Thus, in addition to the 
ACWA examination, 0PM decided to develop rating schedules that could be used by 
agencies to hire at the entry level for professional and administrative occupations. Both the 
ACWA examination and the rating schedules can be delivered via MARS. 

Rating Schedules as an Examining Method 

Although "rating schedule" is a generic term referring to a number of different procedures, 
most Federal rating schedules evaluate an applicant’s relevant training, education, and/or 
experience. Eligible applicants receive ratings from 70 to 100 based on the type and amount 
of training and experience that they possess. 

The Behavioral Consistency Method. The traditional Behavioral Consistency Method 
of training and experience evaluation, originally developed at 0PM (see Schmidt, Caplan, 
Bemis, Decuir, Dunn, & Antone, 1979), asks candidates to describe their major 
achievements in several job-related areas. The job-related dimensions have been identified by 
supervisors as those dimensions that differentiate employees who are performing at superior 
and minimally acceptable levels. The achievements provided by the candidates are evaluated 
using benchmark achievements that have been scaled (i.e., assigned point values) by subject 
matter experts (SMEs). 

Selection methods that assess training and experience attempt to predict future job 
performance from past related accomplishments. A rating (score) is assigned through 
systematic, judgment-based evaluations of information provided by applicants on resumes, 
applications, or other documents (McDaniel, Schmidt, & Hunter, 1988). 



' The opinions expressed in this pt^r are the authors’ and do not necessarily represent the official policy of 
the U.S. Office of Personnel Management. 
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Table 1 



ACWA OCCUPATIONS BEING STUDIED 



Group 1. Health, Safety, and 
Environmental 
Occupations 

Series Title 

00^8 Safety and Occupational Health 
Management 

0023 Outdoor Recreation Planning 
0028 Environmental Protection 
Specialist 

0673 Hospital Housekeeping 
Management 

0685 Public Health Program 
Specialist 

Group 2. Writing r.nd Public 

Information Occupations 

Series Title 

1001 General Arts and Information 
1035 Public Affairs 

1082 Writing and Editing 

1083 Technical Writing and Editing 
1 147 Agricultural Mailcet Reporting 
1412 Technical Information Services 
1421 Archives Specialist 

Group 3. Business, Finance, and 

Management Occupations 

Series Tltie 

0011 Bond Sales Promotion 

0106 Unemployment Insurance 

0346 Logistics Maiugement 

0391 Telecoimnunications 

0501 Financial Admin, and Programs 

0560 Budget Analysis 

0570 Financial Institution Examining 

1101 General Business and Industry 

1 102 Contract Specialist 

1 103 Industrial Property Management 

1104 Property Disposal 
1130 Public Utilities Specialist 
1 140 Trade Specialist 

1 145 Agricultural Program Specialist 

1146 Agricultural Marketing 
1150 Industrial Specialist 

1 160 Financial Analysis 
1 163 Insurance Examining 
1165 Loan Specialist 

1170 Realty 

1171 Appraising and Assessing 
1 173 Housing Management 
1176 Building Management 
1910 Quality Assurance Specialist 
2001 General Supply 

2003 Supply Program Management 
2010 Inventory Management 
2030 Distribution Facilities and 
Storage Management 



Group 3, Business, Finance, and 

Management Occupations 
(continued) 

Series Title 
2032 packaging 
2050 Supply Cataloging 
2101 Transportation Specialist 
2110 Transportation Industry 
Analysis 

2125 Highway Safetv Management 
2130 Traffic Management 
2150 Transportation Operations 

Group 4 , Personnel, Administration, 
and Computer Operations 

Series TiUe 

0142 Manpower Development 
0201 Personnel Management 
0205 Military Personnel Management 
0212 Personnel Staffuig 

0221 Position Classification 

0222 Occupational Analysis 

0223 Salary and Wage Administration 
0230 Employee Relations 

0233 l.abor Relations 
0235 Employee Development 
0244 Labor Management Relations 
Examining 

0246 Contractor Industrial Relations 
0301 Miscellaneous Administration 
aiKi Programs 

0334 Computer Specialist (Trainee) 
0341 Administrative Officer 
0343 Management Program Analysis 
1715 Vocational Rehabilitation 

Group 5. Benefits Review, Tax, and 
Legal Occupations 

Series Title 

0105 Social Insurance Administration 

0187 Social Services 

0270 Civil Service Retirement 

0526 Tax Technician 

0950 Paralegal Specialist 

0958 Pension Law Specialist Series 

0962 Contact Representative 

0965 Land Law Examining 

0967 Passport and Visa Examining 

C/987 Tax Law Specialist 

0990 General Claims Examining 

0991 Worker's Compensation Claims 
Examining 

0993 Railroad Retirement Claims 
Examining 

0994 Unemployment Compensation 
Claims Examining 

0996 Veterans Claims Examining 




''i 



Group 6. Law Enforcement and 

Investigation Occupations 

Series Title 
(X)25 Park Ranger 
0080 Security Administration 
0132 Intelligence 
0249 Wage and Hour Compliance 
1169 Internal Revenue Officer 
1801 Civil Aviation Security 
Specialist 

1810 General Investigator 

1811 Criminal Investigator 

1812 Game Law Enforcement 
1816 Immigration Inspection 
1831 Securities Compliance 

Examining 

1854 Alcohol, Tobacco, and Firearms 
Inspection 

1864 Public Health Quarantine 
Inspection 

1889 Import Specialist 

1890 Customs Inspector 

Group 7. Positions with Positive 
Education Requirements 

Series Tide 

(X)20 Community Planning 
0101 Social Science 
0110 Economist 

0130 Foreign Affairs 

0131 Internal Relations 
0140 Manpower Research and 

Analysis 

0150 Geography 
0170 History 
0180 Psychology 
0184 Sociology 
0190 General Anthropology 
0193 Archaeology 
1015 Museum Curator 
1420 Archivist 

1701 General Education and Training 
1720 Education Program 
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Validity of rating schedules. The validity of rating schedules that assess training and 
experience usually is established using a content validity model. The tasks or behaviors 
measured in the rating schedules are content valid because they sample the kinds of tasks or 
behaviors required in the occupation. Similarly, the competencies measured in the rating 
schedules are content valid because they sample the same competencies required for effective 
performance on the job. 

In addition, there are a few reported studies of the predictive validity of these instruments. 
McDaniel et al. (1988) conducted the most comprehensive review of the validity of methods 
for rating tra ining and experience in personnel selection. The highest validity was found for 
the Behavioral Consistency Method (.45). The Behavioral Consistency Method yields useful 
levels of validity with little variance from study to study, thus, supporting validity 
generalization. Because of its demonstrated level of criterion-related validity, the Behavioral 
Consistency Method was chosen as the model to follow when developing the rating schedules 
for these entry-level occupations. 

Adverse impact^ of rating schedules. Very little evidence is available on the adverse 
impact of rating schedules. Hough (1984) has reported effect sizes of about 1/3 of a standard 
deviation. Thus, rating schedules have less adverse impact than cognitive ability tests, which 
typically show an effect size of approximately one standard deviation. 

Use of Biodata 

The current ACWA examination, which assesses cognitive skills, includes a biodata 
component, the Individual Achievement Record (lAR), which asks questions about an 
applicant’s experience, skills, achievements in school, employment, and other activities. 
Validities of the lAR for predicting job success in ACWA occupations have been well 
documented in a large-scale validity study (Gandy, Dye and MacLane, 1994; Gandy, 
Outerbridge, Sharf, & Dye, 1989). The construct validity of the lAR has also been 
documented (Dye, 1990). In addition, Gandy et al., (1994) found that the adverse impact of 
the lAR is comparable to that typically found for rating schedules. MacLane (1991) recently 
developed a shortened form of the lAR. This inventory was reviewed to identify verifiable 
biodata items for inclusion in the new ACWA rating schedules. 

Thus, the newly developed examining instrument is made up of some of the verifiable 
biodata items that research has shown predict job performance and an application form that is 
based on the Behavioral Consistency Method. The two methods are similar in that both 
assess applicants based on their past accomplishments. In addition, they both produce 
relatively little adverse impact when compared to other selection methods, and past research 
shows that the lAR is valid for ACWA occupations. 



^Technically, adverse impact refers to selection ratios. However, in the personnel psychology literature, it 
often refers to mean differences in performance for groups expressed in standard deviation units. 
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OCCUPATIONAL ANALYSIS 



0PM collected occupational analysis data on each of the 1 10 professional and administrative 
occupations currently covered by the ACWA examination program. Data were collected for 
the entry-level grades for these occupations, typically grades 5 and 7. To do this, 0PM used 
the Multipurpose Occupational Systems Analysis Inventory-Closed-Ended (MOSAIC) 
approach to occupational analysis. MOSAIC is an occupational analysis method of gathering 
data for a variety of human resource management purposes (U.S. Office of Personnel 
Management, 1992). Unlike many other occupational analysis methods, MOSAIC entails 
constructing a survey based on an exhaustive literature review of the target occupations and 
provides results of the occupational analysis in an automated format. 

Literature Review 

The MOSAIC literature review began with acquiring lists of tasks and competencies for the 
occupations under analysis from a wide range of sources, 'fhese sources included task and 
competency documentation from Federal agencies, previous 0PM studies, and well- 
established competency taxonomies such as those from the SCANS (Secretary’s Commission 
on Achieving Necessary Skills) report (SCANS Report for America 2000, 1^2). 

OPM requested task data from the two agencies that employed the most incumbents in each 
occupation for all grade levels. OPM entered all the task information into a database and 
grouped the tasks into general task statements using Alpha Four software (Alpha Software 
Corporation, 1993). 

Concurrent with the development of a task list, OPM constructed a competency list. OPM 
relied on sever. U sources to develop a comprehensive competency list. Two primary sources 
for competencies were previous MOSAIC studies (Gregory, Armitage, Corts, & Park, 1992; 
Rodriguez, Usala, «& Shoun, in progress; U.S. Office of Personnel Management, 1992) and 
the ACWA occupational analysis that was conducted for the ACWA examination in 1990 
(O’Leary, Rheinstein, & McCauley, 1991). The competency lists from these studies were 
supplemented with competencies from other sources, including agencies’ competency lists, 
OPM Position Classification Standards, and the SCANS report. The final ACWA 
competency list is presented in Table 2. 

Psychologists’ Ratings 

Once the task and competency lists were established, OPM psychologists were asked to make 
two ratings. First, they rated each competency for importance to each of the 110 ACWA 
occupations using general occupation descriptions and selected agency position descriptions. 
In addition, they linked the tasks to the competencies necessary to perform the tasks. 
Together these ratings were used to identify which competencies to include on the rating 
schedules. In addition to the psychologists’ ratings, a large-scale occupational analysis survey 
is currently underway which involves collecting task and competency ratings from ACWA 
supervisors and incumbents. This information will be used to revise the rating schedules as 
needed. 
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Table 2 



MOSAIC Competencies; ACWA 



Reading “ Understands and interprets written material, including rules, 
regulations, instnictions, reports, chans, graphs, and tables; applies what 
is learned from written material to specific situations. 

Writing - Uses correct English grammar, punctuation, and spelling to 
communicate facts, ideas, and messages in a succinct and organized 
manner: produces written information that is appropriate for the 
intended audience, 

Arithmetic - Performs computations such as addition, subtraction, 
multiplication, and division correctly using whole numbers, fractions, 
decimals, and percentages. 

Matfaematkal Reasoning - Solves practical problems by choosing 
appropriately from a variety of mathematical and statistical techniques. 

Oral Communication - Expresses ideas and facts to individuals or 
groups effectively, taking into account the audience and nature of the 
information (for example, sensitive, controversial): makes clear and 
convincing oral presentations: listens to others. 

Creative ThMutg - Uses imagination to develop new insights into 
situations and applies innovative solutions to problems: designs new 
methods where established methods and procedures are inapplicable or 
are unavailable. 

Decision Afaidng - Makes sound and well-informed decisions: perceives 
the impact and implications of decisions: commits to action, even in 
uncertain situations, to accomplish organizational goals; causes efumge. 

Reasoning - Discovers or selects rules, principles, or relationships 
between facts and other information; is able to make correct inferences 
from available information. 

Froblem Solving - Identifies and analyzes problems; gathers, interprets, 
and evaluates information to determine its accuracy and relevance: uses 
sound judgment to generate and evaluate alternatives and recommend a 
solution. 

Mental Visualizatioa - Sees diings in die mind by mentally organizing 
and processing symbols, pictures, graphs, objects, or other information 
(for example, sees a building from a blueprint, or secs die flow of work 
activities from reading a work plan). 

Learning - Uses efficient learning techniques to acquire and apply new 
knowledge and skills, 

Self-Esteem - Believes in own self-worth; maintains a positive view of 
self and displays a professional image. 

Teamvrork - Encourages arui facilitates cooperation, pride, trust, and 
group identity: fosters commitment and team spirit: works with others to 
achieve goats, 

Integrity/Honesty - Displays high standards of ethical conduct and 
understands die impact of violating these standards on an organization, 
self, and others; chooses an ediical course of action; is trustworthy. 

Self-Management • Sets well-defined and realistic personal goals: 
displays a high level of initiative, effort, and comnutment towards 
completing assignments in a timely manner with minimal supervision; is 
motivated to achieve; demonstrates responsible behavior, 

InUrpersanal Skills • Shows understanding, friendliness, courtesy, tact, 
empathy, concern, and politeness to others: relates well to people from 
varied backgrounds and different situations: is sensitive to cultural 
diversity, race, gender, and other individual differences in the workforce. 



Planning and Evaluating - Organizes work, sets priorities, and 
determines resource requirements; determines objectives and strategies 
to achieve them; monitors and evaluates progress against goals. 

Financial ManagemetU - Prepares, justifies, and/or admitdsters the 
budget for program areas; plans, administers, and monitors expenditures 
to ensure cost-^ective support of programs and policies. 

Managing Human Resources • Plans, distributes, and monitors work 
assignments; evaluates work performance and provides feedback to 
others on their performance. 

Leadership - Interaas wish others to influence, motivate, and challenge 
them. 

Teaching Others - Helps others learn; identifies training needs: provides 
constructive feedback: coaches others on how to perform tasks: acts as 
a mentor. 

Customer Service - Works and comnuuucates with clients and customers 
(that is, any individuals who use or receive the services or products that 
your work unit produces, including the general publi< , individuals who 
work in your agency, other agencies, or organiz0ions outside the 
Government) to provide informaHon and satisfy their expeaations: 
committed to quality services. 

Organizational Awareness - Knows how social, political, 
organizational, and technological systems work and operates effectively 
widiinthem; this includes die policies, procedures, rules, and regulations 
of the work or organization. 

IirfUieHcing/NegodadHg - Persuades others to accept recommendations: 
works With others towards an agreement: negotiates to find mutually 
acceptable solutions. 

Technology /^pUcadon - Uses computers and computer applications (for 
example, spreadsheet, word processing, database management, and 
graphics software) to analyze and communicate information in the 
appropriate format (for example, line graphs, bar charts, pie charts, 
tables, narrative); uses technology to work more efficiently and improve 
work processes and products. 

Flexibility - Is open to change and new information; adapts behavior and 
work methods in response to new information, changing conditions, or 
unexpected obstacles: effectively deals with pressure and ambiguity. 

Technical Competence - Uses knowledge that is acquired through 
formal training or extensive on-the-job experience to perform one*s job; 
works with, understands, and evaluates technical material related to the 
job. 

Perceptual Speed - Sees detail in words, numbers, pictures, and graphs, 
quickly and accurately. 

Pfayakal Strength and Agility - Ability to bend, lift, climb, stand, or 
walk for long periods of time; ability to exert oneself physically over 
time; ability to perform moderately laboring work. 

Memory - Recalls information that has been presented previously. 

Eye-Hand Coordinatioa - Accurately coordinates one*s eyes widi one's 
fingers, wrists, or arms to move, cany, or manipulate objects, or to 
peiform odier job-related tasks. 

Vision - Understands where die organization is headed and how to make 
a contribution; recognizes opportunities to help die organization 
accomplish its objectives. 
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RATING SCHEDULE DEVELOPMENT 



The Behavioral Consistency Method was used to develop scaled accomplishments suitable for 
use across the ACWA occupations. Accomplishments were collected for the competencies rated 
as important by 0PM psychologists and were scaled by supervisors of ACWA occupations. A 
ra ting schedule was developed for each occupation that included accomplishments related to the 
competencies rated as important for each occupation. Items from the lAR were also included on 
each occupation-specific rating schedule. 

To facilitate automation efforts, the rating schedules were designed in a self-rating format. Mabe 
and West (1981), in an examination of various factors affecting self-ratings, suggested that the 
most important factor contributing to the validity of self-ratings is the expectation that the self- 
ratings will be checked objectively. Thus, the rating schedules include several features to 
indicate the responses will be checked: (1) a general statement is included on the application 
form stating that misrepresentation can lead to dismissal, (2) applicants are required to submit 
a resumd or similar information along with their applications that can be used to verify their 
responses to questions on the rating schedule, and (3) applicants are told that the selecting 
official may review the information provided on the rating schedule during the selection 
interview. 

The major tasks and steps that were followed to develop rating schedules for each ACWA 
occupation are described below. 

Task 1: Accomplishments Database Development 

Step 1; Identify rating competencies to assess with the rating schedule method. The 
competencies identified in the occupational analysis literature review, along with the 
competencies currently assessed by the ACWA examination, were considered for use on the 
rating schedules. During the occupational analysis, 0PM psychologists rated how important each 
competency was for successful performance in each ACWA occupation. These ratings were 
reviewed to identify competencies that were consistently rated as important across occupations. 
These competencies were retained for further consideration. 

The remaining competencies were reviewed for feasibility of assessment. A panel of 0PM 
personnel research psychologists experienced in rating schedule development identified 
competencies that were suitable for assessment by the rating schedule method. This final set of 
competencies, the rating competencies, was the target of subsequent rating schedule development 
activities. These ratings are shown in italics in Table 2. 

Step 2: Collect accomplishments. An Accomplishments Record Fom was developed for 
collecting accomplishments for the rating competencies from ACWA applicants. The Form was 
distributed to ACWA applicants who provided accomplishments on a voluntary basis during May 
and June of 1994. The applicants were informed that the accomplishments they provided were 
being collected for test development purposes and would not be scored. They were instructed 
to provide accomplishments from their educational experience, work experience, and volunteer 
work. Over 3,000 accomplishments were collected on 17 competencies. 
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Task 2: Scaling of Accomplishments 

Step 3: Develop Accomplishments Survey. The accomplishments collected in Step 2 
were reviewed by 0PM psychologists and rewritten in a general form that would represent 
specific, but related, accomplishments. Redundant accomplishments were combined or 
eliminated. Each accomplishment was stated so that it could be used for all ACWa occupations. 
The revised accomplishments were formatted as an Accomplishments Survey for SMEs to use in 
classifying and scaling the accomplishments. 

Step 4: Collect SME accomplishment scaling data. Subject matter expert panels, 
consisting of supervisors of ACWA incumbents, were formed to provide accomplishment scaling 
data. The supervisors convened at the 0PM regional offices, and 0PM psychologists facilitated 
the groups in completing the Accomplishments Survey. 

The group facilitator helped familiarize the supervisors with the competencies and their 
definitions. First, the supervisors were instructed to indicate whether each accomplishment was 
representative of the competency to which it had been assigned. Next, they were instructed to 
assign each accomplishment a value of high, medium, or low, indicating whether the 
accomplishment demonstrated a high, medium or low level of proficiency on that competency. 
Finally, they were asked to provide feedback on the clarity and appropriateness of the 
accomplishments . 

Task 3: Rating Schedule Assembly 

Step 5: Identify accomplishments to include on rating schedules. The accomplishment 
scaling data provided by the supervisors were analyzed to identify accomplishments classified 
at the high, medium, and low proficiency levels for each competency. 0PM psychologists also 
used the proficiency ratings to determine which accomplishments could be combined, based on 
similar content and proficiency level ratings, and which accomplishments could be eliminated, 
either because they were not representative of the competency or because they received very low 
proficiency level ratings (meaning the applicant should not receive credit for performing that 
low-level accomplishment) . Accomplishments retained for each competency represent educational 
experience, work experience, volunteer work, and, in some cases, personal or home-related 
experiences. 

Step 6: Assemble and review occupation-specific rating schedules. The occupational 
analysis data were used to determine which competencies to include on the rating schedules for 
each occupation. A rating schedule was assembled for each occupation that included instructions 
for recording responses on a scannable answer sheet, minimum qualification items, selected lAR 
items, and scaled accomplishments associated with the competencies identified as important for 
that occupation by the occupational analysis. 
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OPERATIONAL IMPLEMENTATION 



MARS is an integral part of the implementation of the ACWA rating schedules. MARS was 
developed to provide 0PM Service Centers and Federal agencies with an efficient and flexible 
system for examining and rating applicants that also reduces staff time and costs. The system 
can be used to design rating schedules, scan responses, rate applicants, and gent“ate a list of 
eligible candidates. The components required for this processing are a personal computer and 
an optical scanner. 

Because of the automated format, applicants had to be able to respond to the rating schedule 
using a scannable form. Thus, the accomplishment items were worded so that applicants crnld 
respond either "yes" or "no" to each item. This is a significant departure from the traditional 
Behavioral Consistency Method, which requires applicants :0 provide open-ended responses 
describing their accomplishments for job-related competencies. 

The instructions and randomized rating schedule items were entered into MARS to create a 
rating schedule for each of the 110 ACWA occupations. In addition, point values for the 
accomplishment items were entered based on the weights assigned by the supervisors. 
Transmutation tables were developed to convert the raw scores to the 70 to 100 point range 
traditionally used in Federal ratings. (Additional points are added for veterans’ preference.) 

Potential applicants responding to a vacancy announcement receive the Supplemental 
Qualifications Statement (rating schedule) which includes background items, minimum 
qualifications items, lAR items, and accomplishment items, along with a scannable response 
form. Applicants are also required to submit a resum6 or a standard application form, which can 
be used to verify their responses to the rating schedule items. 

VALIDITY STRATEGIES 

Discussions of the types of validity suggest that constmct validity is the umbrella validity under 
which other traditional forms of validity fall (Arvey, Nutting, & Landon, 1992; Landy, 1986; 
Tenopyr, 1977). Thus, a nmnber of different, but complementary, validity strategies are being 
used to validate the rating schedules. 

Meta-Analytic Studies 

Meta-analytic studies of the traditional Behavioral Consistency Method have shown that the 
procedure produces instruments with useful levels of validity. Moreover, the fact that the 
distributions of validity coefficients have small corrected standard deviations supports validity 
generalization (McDaniel et al., 1988). Although the number of studies in the database is 
somewhat limited, the studies do provide historical support for the validity of tlie procedure. 
Meta-analytic studies of biodata support validity generalization of this selection procedure as well 
(Rothstein, Schmidt, Erwin, Owens, & Sparks, 1990). 
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Content Validity 



Rating schedules are typically validated using a content validity model. Schmidt et al. (1979) 
indicated that a content validity strategy is applicable to the development and validation of an 
examination using the Behavioral Consistency Method. The behaviors sampled in the 
accomplishments are content valid because they sample the kinds of achievements required to 
perform in the occupation. Schmidt et al. stated that the purpose of developing competencies is 
to point out and direct attention to the areas in the background of the applicants where the most 
valid accomplishments are likely to be found. 

The Society for Industrial-Organizational Psychology’s Principles for the Validation and Use of 
Personnel Selection Procedures (1987) support the use of content validity for accomplishments. 
Accomplishments can be justified by a similarity between the content of the personal history 
experience and the content of the occupation, regardless of whether or not the personal history 
experience and occupation as a whole are similar. As indicated in the Uniform Guidelines on 
Employee Selection Procedures (Federal Register, 1978), this relationship can be based on the 
similarity between the competencies or behaviors documented in the accomplishments and those 
of the occupation rather than between the exact content of the accomplishments and the job. For 
instance, a person with budgetary experience need not demonstrate experience with the specific 
budgetary items associated with an occupation for the accomplishments to be content valid, 
provided the competencies or behaviors in the accomplishments are similar to those required by 
the job. The rating sc'iedule development process was based on the content validity model. Data 
from the psychologists’ ratings (both the competency importance ratings and the task-competency 
linkages) and the large-scale occupational analysis currently underway will be used to provide 
evidence of the content validity of the rating schedules. 

Criterion-related Validity 

Criterion-related validity studies will be conducted for rating schedules of occupations that have 
a large number of incumbents or for occupations where a large number of hires are anticipated. 
The first of these studies will be conducted in late 1995. Additional studies will be conducted 
on an ongoing basis. The approach for these studies will be similar to the approach used to 
validate the ACWA examination (see Diane & MacLane, 1994; Pollack & Paskey, 1993; Reilly, 
1993a; Reilly, 1993b; Reilly, Nester, McGilvray, & Kelly, 1991). For occupations where small 
number of hires are the norm, an alternative validation strategy will be used. This strategy will 
involve validating each competency across occupations rather than assessing the validity of the 
rating schedule as a whole for a particular occupation. 

First, a study of the technical feasibility of conducting such studies will be completed (see 
Anderson, 1988). Once the feasibility of the study is established, applicants or current 
incumbents will be asked to complete the rating schedule for a particular ACWA occupation. 
Scores on the rating schedule will be correlated with various performance measures, including 
supervisory performance evaluations, job knowledge tests, and/or work samples. The criterion 
measures will be developed to cover both cognitive and noncognitive performance dimensions 
to accommodate the range of competencies being assessed by the ACWA selection procedures. 
Because the dimensions included on the rating schedules are based on a thorough occupational 
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analysis, 0PM anticipates that the correlations between rating schedule scores and performance 
in training or on the job will be significant. Fairness analyses will also be conducted at this time. 

Construct Validity 

While the construct foundation of cognitive ability tests is well established (Northrop, 1989), no 
systematic research has been conducted to establish the construct validity of rating schedules and 
the attributes measured by them. The development of the rating schedules for the ACWA 
occupations provides us with a unique opportunity to establish evidence concerning the construct 
validity of dimensions commonly assessed with rating schedules. 

In the employment context, construct validity has traditionally been demonstrated by identifying 
(through duty-ability linka'^e in occupational analysis) the abilities that are necessary for 
successful performance on trie job. Once these abilities are identified, measures of such abilities 
are chosen or developed. Next, evidence is advanced that the measures chosen or developed are 
indeed measures of the identified job-relevant abilities. 

In the case of the ACWA examination, over six decades of psychometric and cognitive research 
support the psychometric viability of the fundamental abilities measured by the examination (see 
Northrop, 1989). Such evidence is lacking for many of the attributes measured by traditional 
rating schedules, even though they are one of the most frequently used selection devices in the 
Federal Government. 

We propose to do a series of studies, similar to those undertaken in the construct validation of 
the lAR (see Gandy et al., 1994). Construct validity will be assessed through the use of marker 
tests which have known psychometric properties. We will hypothesize the relationship between 
the attributes measured in the rating schedules and the marker tests, and then conduct empirical 
studies to support or refute these hypotheses. The content and criterion-related validity studies 
discussed earlier will provide additional construct validity evidence. 

SUMMARY 

These new ACWA hiring procedures provide agencies with another option for hiring at the entry 
level for professional and administrative occupations. They also allow agencies to assess 
competencies that tre not currently assessed by the ACWA examination (such as interpersonal 
skills, oral communication, customer service). Because the rating schedules had to be automated, 
an innovative approach was used to develop training and experience items that could be 
answered on a scannable answer sheet. The automated format reduces the burden on both the 
applicants, who do not have to provide lengthy written accounts of their traming and experience, 
and personnel specialists, who do not have to manually rate and rank applicants. 
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